Evaluating the Dimensionality of the 2001 PIRLS Reading Assessment: An application of DIMTEST with DESM and CFA

The purpose of this study was to investigate the dimensionality of the reading assessment of 2001 Progress in International Reading Literacy Study (PIRLS) by DIMTEST with DIMTEST-based Effect Size Measure (DESM) and Confirmatory Factor Analysis (CFA). The 2001 PIRLS reading assessment was comprised of eight booklets, with accompanying items based on different reading passages. With respect to the potential problem of reading passage dependency on measurement precision and content-related validity in reading comprehension tests, the study examined the dimensionality of the reading assessments by booklet of reading passage; a total of ten pairs of booklets were tested. As implied in previous studies reporting unexpected high Type I error rate of DIMTEST (Froliech & Habing, 2003; Seo & Roussos, 2006; Stout, Froelich, & Gao, 2001), DIMTEST rejected the null for all the ten cases; A pair of booklets is not dimensionally similar. DESM used in tandem with DIMTEST resulted in six sets that were unidimensional, but the other four sets were not. Despite different underlying assumptions, several pairs were found to be unidimensional, which is common in DESM and CFA.